In the field of software engineering, performance testing is often performed during the development of a system or a software application, to determine how fast some aspect of that system or that software application performs under a particular workload. The testing can be used to validate and verify other attributes of the system, such as the system's scalability, or reliability. Performance testing can also serve a variety of end purposes: for example it can be used to compare two differently-configured systems to find which configuration performs better; or it can measure which parts of the system or which types of workload are likely to cause the system to perform badly. Performance testing can also be used, sometimes from a marketing perspective, to allow a software vendor to demonstrate that their particular system or software application meets (or fails to meet) the performance criteria or expectations set by a group of customers. With respect to a customer's objective and subjective expectations, performance testing may also demonstrate how well, or how poorly, the system performs against those expectations, and can provide a prospective customer with information they can then use to decide which system, or system configuration, is most likely to suit their needs.
Generally speaking, performance testing technology emulates the typical operating environment of the system or software application; and then often includes tests that measure the extremes of the environment that the system or the software will be operating within. Typically, a variety of tests are conducted and coordinated, while test values (metrics) and other performance data are gathered. These values, metrics, performance data, and other results are then analyzed to give a sense of the performance of the system.
In the field of application servers, and other server-related technologies, performance tests are mostly used by vendors to determine runtime characteristics, i.e. they are used as a measure of how the underlying system will perform at runtime at a particular customer's work site. In this context performance tests are very useful, and examples of such tests and benchmarks include “how many pages can be served up by the server in a particular amount of time?”.
These types of performance test generally work on “standard applications”. But they are not readily used with non-standard applications, nor are they readily used in an integrated development environment (IDE) or many other software development environments. Currently there is no good measure that can be used in the computer industry to signify how an IDE will perform. While some application server customers may desire certain characteristics as goals in their IDE environment, it is difficult to quantify these goals. Furthermore since it is difficult to quantify, it is similarly difficult for the software developer to develop an IDE suite that addresses those customer desires. This is the area the present invention is designed to address.